Using specularities in comparing 3D models and 2D images

We aim to create systems that identify and locate objects by comparing known, 3D shapes to intensity images that they have produced. To do this we focus on verification methods that determine whether a known model in a specific pose is consistent with an image. We build on prior work that has done this successfully for Lambertian objects, to handle a much broader class of shiny objects that produce specular highlights. Our core contribution is a novel method for determining whether a known 3D shape is consistent with the 2D shape of a possible highlight found in an image. We do this using only a qualitative description of highlight formation that is consistent with most models of specular reflection, so no specific knowledge of an object’s specular reflectance properties is needed. This allows us to treat non-Lambertian image effects as a positive source of information about object identity, rather than treating them as a potential source of noise. We then show how to integrate information about highlights into a system that also checks the consistency of Lambertian reflectance effects. Also, we show how to model Lambertian reflectance using a reference image, rather than albedos, which can be difficult to measure in shiny objects. We test each aspect of our approach using several different data sets. We demonstrate the potential value of our method of handling specular highlights by building a system that can locate shiny, transparent objects, such as glassware, on table tops. We demonstrate our hybrid methods on pottery, and our use of reference images with face recognition experiments. ? A more preliminary version of this work appears as: [36] ∗ Corresponding author. 1 Computer Science Department, University of Haifa, Mount Carmel, Haifa 31905, Israel. e-mail: [email protected] 2 Computer Science Department, University of Maryland, College Park, Maryland. e-mail:[email protected] and [email protected] 3 Computer Science Department, Columbia University,450 Computer Science Bldg 500 W 120 St New York, NY 10027. e-mail:[email protected] Preprint submitted to Elsevier 28 May 2008 Fig. 1. Images of real objects contain effects due to specularities, transparency, and interreflections.